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Remarks 

This Reply is in response to the Office Action mailed February 9, 2010. 

I. Summary of Examiner's Rejections 

In the Office Action mailed February 9, 2010, Claims 1-5 and 9 were rejected under 35 
U.S.C. 103(a) as being unpatentable over Ludwig et al. (U.S. Patent No. 7,185,054, hereafter 
Ludwig) in view of Ayatsuka et al. (U.S. Patent No. 7,188,139, hereafter Ayatsuka). Claims 6,8, 10- 
13, 16-17 and 37-46 were rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig in 
view of Ayatsuka, and further in view of Nelson et al. (U.S. Patent Publication No. 2004/0236830, 
hereafter Nelson). Claim 7 was rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig 
in view of Ayatsuka, and further in view of Emens et al. (U.S. Patent No. 6,463,343, hereafter 
Emens). Claims 18 and 21 were rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ludwig in view of Nelson. Claims 19-20 and 22-23 were rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig in view of Nelson, and further in view of Ayatsuka. Claims 24, 26-27, 30, 
32, and 34-35 were rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of 
Ayatsuka, and further in view of Burt et al. ("Object tracking with a moving camera," IEEE An 
Application of Dynamic Motion Analysis, 1989, pp 2-12, hereafter Burt) and Nelson. Claim 25 was 
rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of Ayatsuka and Burt, 
and further in view of Andersson (U.S. Patent Publication No. 2002/0111999 A1). Claim 28 was 
rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of Ayatsuka, Burt, and 
Nelson, and further in view of Hildebrandt (U.S. Patent Publication No. 2004/0070616). Claim 29 
was rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of Ayatsuka, Burt, 
Nelson, and Andersson, and further in view of Westfield (U.S. Patent No. 6,677,979). Claim 31 was 
rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of Ayatsuka, Burt, and 
Nelson, and further in view of Westfield. 

II. Summary of Applicant's Amendments 

The present Reply amends Claims 1, 10, 12, 13, 17,18, 24, 37 and 46, leaving for the 
Examiner's present consideration Claims 1-13, 16-32, 34-35, and 37-46. Reconsideration of the 
Application, as amended, is respectfully requested. 

III. Claim Rejections Under 35 U.S.C. § 103 
Claims 1-13. 16-17 and 37-46 
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In the Office Action mailed February 9, 2010, Claims 1-5 and 9 were rejected under 35 
U.S.C. 103(a) as being unpatentable over Ludwig (U.S. Patent No. 7,185,054) in view of Ayatsuka 
(U.S. Patent No. 7,188,139). Claims 6, 8, 10-13, 16-17 and 37-46 were rejected under 35 U.S.C. 
103(a) as being unpatentable over Ludwig in view of Ayatsuka, and further in view of Nelson (U.S. 
Patent Publication No. 2004/0236830). Claim 7 was rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig in view of Ayatsuka, and further in view of Emens (U.S. Patent No. 
6,463,343). 

Claim 1 

Claim 1, as amended, recites: 

1. (Currently Amended) A method for exchanging information in a shared interactive 
environment, comprising: 

selecting a first remote physical device in a first live video image that shows a first 
view of the shared interactive environment, wherein information is associated with the first 
remote physical device; 

causing the information to be transferred to a second remote physical device shown 
in a second live video image that shows a second view of the shared interactive 
environment, wherein the transfer is brought about by 

manipulating a visual representation of the information shown in the first live 
image by interacting with the first live video image and the second live video image, 

querying the second remote physical device to determine if it can receive the 
information from the first remote physical device, and 

transferring the information from the first remote physical device to the 
second remote physical device; 

wherein at least one of the first remote physical device and the second remote 
physical device has a statically or dynamically defined hotspot in the first live video image or 
the second live video image; 

wherein the first remote physical device and the second remote physical device are 
part of the shared interactive environment; and 

wherein the first remote physical device and the second remote physical device are 
different remote physical devices. 

Ludwig discloses a teleconference system for conducting a teleconference among a plurality 
of participants. The system has a plurality of video display devices, each having associated 
participant video capture capabilities and participant audio capture and reproduction capabilities. 
(Abstract). 
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Ayatsuka discloses that "Hyper-drag" provides a user interface environment for the portable 
user computer to cooperate with a shared computer designed to use a desktop or a wall as display 
screen. For example, "hyper-drag" operations can be supported by introducing a camera-based 
object recognition system into the information environment. More specifically, the image of a 
computer display screen may be projected onto the surface of a disk top [sic] or a wall by means of 
a projector. Furthermore, both a projected image that is an object in the logical space of the 
computer and an object of the real world (that may be found on a disk top [sic]) may be recognized 
on the basis of a corresponding image picked-up by a camera and the motions of the objects may 
be traced so that the user can operate the computer interactively on the display screen that is now 
expanded to the disk top [sic] or the wall. (Column 3, lines 49-63). If, on the other hand, the target 
is a device that can display transparencies, the transparency is transferred to the projector (Step 
S107) in response to the request of the user of the user terminal 10 for transferring the current 
transparency to the projector (Step S106). A request for transferring a transparency may be issued 
in an intuitive way by using, for instance, "hyper-drag" and expanding the current GUI operation into 
the space of the real world. Transparencies are transferred by way of the network 70. After the 
completion of the transfer of data, the processing operation returns to Step S101 and repeats the 
above steps. (Column 21, lines 7-17). 

As described above, Applicant respectfully submits that in Ludwig a teleconference system 
can show video of several participants and that these participants can interact with one another by 
annotating a graphical image shown by the teleconference system. 

Applicant respectfully submits that Ayatsuka appears to disclose a system that can utilize 
"hyper-drag" which appears to enable a user to click and drag an object shown on a computer 
desktop to a physical surface such as the table on which the computer is placed or an adjacent 
wall. When the user completes such a click and drag operation, an image of the object appears to 
be projected onto the physical surface. Additionally, Ayatsuka appears to disclose that slides can 
be dragged from a computer desktop to a target device such as a projector which can then display 
the slides. 

Claim 1, as amended, recites causing the information to be transferred to a second remote 
physical device shown in a second live video image that shows a second view of the shared 
interactive environment, wherein the transfer is brought about by manipulating a visual 
representation of the information shown in the first live image by interacting with the first live video 
image and the second live video image. 

Applicant respectfully submits that in Ayatsuka, "hyper-drag" does not appear to be used to 
transfer information from one target device to another. Although Ayatsuka appears to show that 
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information can be allocated from one desktop computer to several target devices, Applicant 
respectfully submits that Ayatsuka does not appear to disclose or render obvious transferring 
information directly from one target device to another target device. Additionally, Ludwig does not 
appear to disclose or render obvious transferring information between remote physical devices 
shown in live video images. 

Furthermore, Applicant respectfully submits that in Ludwig the various participants do not 
manipulate the live images of any of the other participants. Similarly, in Ayatsuka no live images 
appear to be manipulated. Instead, "hyper-drag" appears to extend a computer desktop to physical 
surfaces, but the user is limited to manipulating icons (or visual representations of physical objects) 
shown on a desktop. Applicant respectfully submits that neither Ludwig nor Ayatsuka, when 
considered alone or in combination, disclose or render obvious manipulating a visual representation 
of the information shown in the first live image, as recited by Claim 1, as amended. 

In view of the above comments, Applicant respectfully submits that Claim 1, as currently 
amended, is neither anticipated by nor obvious in view of the cited references, and reconsideration 
thereof is respectfully requested. 

Claim 10 

The remarks provided above with regard to the rejection of Claim 1 under 35 U.S.C. 103(a) 
over Ludwig in view of Ayatsuka are herein incorporated by reference. 
Claim 10, as amended, recites: 

10. (Currently Amended) A method for exchanging information in a shared interactive 
environment, comprising: 

selecting a first object wherein the first object is one of 1) a first physical device at a 
remote location shown in a live video image, and 2) an icon on a computing device, wherein 
the live video image shows a view of the remote location which includes a plurality of 
physical devices including the first physical device; 

causing information associated with the first object to be transferred to a second 
object wherein the second object is the other of 1) the first physical device at the remote 
location shown in the live video image, and 2) the icon on the computing device; 

annotating the view of the remote location shown in the live video image with an 
annotation; 

automatically transferring the annotation to the first physical device if the annotation 
is at least partially drawn over the first physical device as it appears in the live video image; 

displaying the annotation on the first physical device such that the annotation can be 
viewed at the remote location; 
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wherein the first physical device has a dynamically defined hotpot in the live video 

image; 

wherein the transfer is brought about by manipulating a visual representation of the 
information in the live video image; 

wherein manipulating includes interacting with the first object in the live video image 
and the second object; and 

wherein the first physical device is part of the shared interactive environment. 

Ludwig discloses a teleconference system for conducting a teleconference among a plurality 
of participants. The system has a plurality of video display devices, each having associated 
participant video capture capabilities and participant audio capture and reproduction capabilities. 
(Abstract). 

Ayatsuka discloses that "Hyper-drag" provides a user interface environment for the portable 
user computer to cooperate with a shared computer designed to use a desktop or a wall as display 
screen. For example, "hyper-drag" operations can be supported by introducing a camera-based 
object recognition system into the information environment. More specifically, the image of a 
computer display screen may be projected onto the surface of a disk top [sic] or a wall by means of 
a projector. Furthermore, both a projected image that is an object in the logical space of the 
computer and an object of the real world (that may be found on a disk top [sic]) may be recognized 
on the basis of a corresponding image picked-up by a camera and the motions of the objects may 
be traced so that the user can operate the computer interactively on the display screen that is now 
expanded to the disk top [sic] or the wall. (Column 3, lines 49-63). If, on the other hand, the target 
is a device that can display transparencies, the transparency is transferred to the projector (Step 
S107) in response to the request of the user of the user terminal 10 for transferring the current 
transparency to the projector (Step S106). A request for transferring a transparency may be issued 
in an intuitive way by using, for instance, "hyper-drag" and expanding the current GUI operation into 
the space of the real world. Transparencies are transferred by way of the network 70. After the 
completion of the transfer of data, the processing operation returns to Step S101 and repeats the 
above steps. (Column 21, lines 7-17). 

Nelson discloses an annotation management system for providing real-time annotations for 
media content during a videoconference session. (Abstract). FIG. 8 is a simplified schematic 
diagram of a conference room configuration in which video conference participants view a video 
conference session from a liquid crystal display (LCD) projector in accordance with one 
embodiment of the invention. Here, in order to support annotation capabilities for participants in the 
conference room the system will support small devices, such as a pocket personal computer 
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connected wirelessly to the network running a small device annotation client software. (Paragraph 
[0067]). FIG. 9 is a flow chart diagram illustrating the method operations for providing real-time 
annotation data to clients of a video conference session in accordance with one embodiment of the 
invention. The method initiates with operation 220 where a display region of a user interface 
associated with a client of the video conference session is annotated. Here, a participant of the 
video conference session may annotate a display region through the use of a mouse, stylus, or 
some other input device in order to highlight, distinguish or somehow otherwise annotate the 
display region. The method then advances to operation 222 where the annotation of the display 
region is detected. For example, a client monitor or some similar functionality, as mentioned above, 
may detect the annotation of the display region. The method then proceeds to operation 224 where 
in response to detecting the annotation of the display region, data corresponding to the annotation 
of the display region is communicated to other clients of the videoconference session. Here, the 
back channel as discussed with reference to FIG. 1, is used to communicate the annotation of the 
display region to the media transport server which in turn communicates the annotation data to the 
annotation management system described with reference to FIG. 2. Accordingly, the real-time 
presentation of the annotation data is capable of being viewed by each participant of the 
videoconference session. (Paragraph [0068]). 

In the Office Action, it was asserted that Nelson discloses participants of a conference may 
use annotation software installed on an electronic device to annotate a live view of a physical 
device (figure 8 display screen 216). The annotation will appear to other conference participants 
and will be projected onto display 216. 

Claim 10 has been amended to more clearly recite annotating the view of the remote 
location shown in the live video image with an annotation; and automatically transferring the 
annotation to the first physical device if the annotation is at least partially drawn over the first 
physical device as it appears in the live video image. 

Thus, in Claim 10, the view of the remote location shown in the live video image is 
annotated and if the annotation is at least partially drawn over the first physical device, then the 
annotation is automatically transferred to that device. As described above, in Nelson the users 
appear to annotate a display region that is shown on a projector. The display region does not 
appear to show a remote location or any physical devices. Instead, the display region in Nelson 
appears to show the content being annotated. Furthermore, in Nelson, annotations appear to be 
displayed via the projector regardless of whether the annotations overlap any physical device 
shown in a live video image. Additionally, as described above with respect to Claim 1, neither 
Ludwig nor Ayatsuka appear to disclose annotating a live video image. As such, Applicant 

- 15- 

Attorney Docket No.: FXPL-01063US0 
M:\nfeld\wp\FXPL\1063US0\1063US0_RespOA_020910.doc 



Application No. 10/629,403 

Reply to Office Action dated: February 9, 2010 

Reply dated: July 9, 2010 

respectfully submits that Ludwig, in view of Ayatsuka and Nelson, does not disclose or render 
obvious annotating the view of the remote location shown in the live video image with an 
annotation; and automatically transferring the annotation to the first physical device if the annotation 
is at least partially drawn over the first physical device as it appears in the live video image, as 
recited by Claim 1, as amended. 

Claim 10, as amended, further recites selecting a first object wherein the first object is one of 
1) a first physical device at a remote location shown in a live video image, and 2) an icon on a 
computing device, wherein the live video image shows a view of the remote location which includes 
a plurality of physical devices including the first physical device. 

As described above, Ludwig appears to show a video conference system which shows live 
views of several people participating in the video conference; Ayatsuka appears to show a hyper 
drag technique that can be used to extend the boundaries of a computer desktop to adjacent 
physical surfaces; and Nelson appears to show an annotation system which enables users to 
annotate a display view. However, Applicant respectfully submits that Ludwig, in view of Ayatsuka 
and Nelson, do not appear to disclose or render obvious that the live video image shows a view of 
the remote location which includes a plurality of physical devices including the first physical device, 
as recited by Claim 1, as amended. Indeed, while Ludwig appears to show live views of 
participants, these appear to be close ups of the participants' faces and do not appear to show any 
physical devices. Additionally, neither Ayatsuka nor Nelson appear to show live video images. 

In view of the above comments, Applicant respectfully submits that Claim 10, as currently 
amended, is neither anticipated by nor obvious in view of the cited references, and reconsideration 
thereof is respectfully requested. 

37 and 46 

The comments provided above with respect to Claim 1 are hereby incorporated by 
reference. Claims 37 and 46 have been similarly amended to more clearly recite the embodiments 
therein. For similar reasons as provided above with respect to Claim 1, Applicant respectfully 
submits that Claims 37 and 46, as amended, are likewise neither anticipated by, nor obvious in view 
of the cited references, and reconsideration thereof is respectfully requested. 

Claims 2-9, 11-17, and 38-45 

Claims 2-9, 11-17, and 38-45 depend from and include all of the features of Claims 1, 10, or 
37. Claims 2-9, 11-17, and 38-45 have not been addressed separately herein; however, Applicant 
respectfully submits that these claims are allowable at least as depending from an allowable 
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independent claim, and further in view of the amendments to the independent claims, and the 
comments provided above. Reconsideration thereof is respectfully requested. 

Claims 18-23 

In the Office Action mailed February 9, 2010, Claims 18 and 21 were rejected under 35 
U.S.C. 103(a) as being unpatentable over Ludwig in view of Nelson. Claims 19-20 and 22-23 were 
rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig in view of Nelson, and further 
in view of Ayatsuka. 

Claim 18 

The remarks provided above with regard to the rejection of Claim 10 under 35 U.S.C. 103(a) 
over Ludwig in view of Ayatsuka and Nelson are herein incorporated by reference. 
Claim 18, as amended, recites: 

18. (Currently Amended) A method for annotating a live video image, comprising: 

displaying a live video image, wherein the live video image shows a view of a 
location including one or more physical devices at the location; 

annotating the view of the location shown in the live video image with an annotation; 

automatically transferring the annotation to one of the physical devices at the 
location shown in the live video image if the annotation is at least partially drawn over the 
physical device as it appears in the live video image; 

displaying the annotation on the physical device such that the annotation can be 
viewed at the location; and 

wherein the annotation is visible to at least one participant in a shared interactive 
environment. 

Ludwig discloses a teleconference system for conducting a teleconference among a plurality 
of participants. The system has a plurality of video display devices, each having associated 
participant video capture capabilities and participant audio capture and reproduction capabilities. 
(Abstract). 

Nelson discloses an annotation management system for providing real-time annotations for 
media content during a videoconference session. (Abstract). FIG. 8 is a simplified schematic 
diagram of a conference room configuration in which video conference participants view a video 
conference session from a liquid crystal display (LCD) projector in accordance with one 
embodiment of the invention. Here, in order to support annotation capabilities for participants in the 
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conference room the system will support small devices, such as a pocket personal computer 
connected wirelessly to the network running a small device annotation client software. (Paragraph 
[0067]). FIG. 9 is a flow chart diagram illustrating the method operations for providing real-time 
annotation data to clients of a video conference session in accordance with one embodiment of the 
invention. The method initiates with operation 220 where a display region of a user interface 
associated with a client of the video conference session is annotated. Here, a participant of the 
video conference session may annotate a display region through the use of a mouse, stylus, or 
some other input device in order to highlight, distinguish or somehow otherwise annotate the 
display region. The method then advances to operation 222 where the annotation of the display 
region is detected. For example, a client monitor or some similar functionality, as mentioned above, 
may detect the annotation of the display region. The method then proceeds to operation 224 where 
in response to detecting the annotation of the display region, data corresponding to the annotation 
of the display region is communicated to other clients of the videoconference session. Here, the 
back channel as discussed with reference to FIG. 1, is used to communicate the annotation of the 
display region to the media transport server which in turn communicates the annotation data to the 
annotation management system described with reference to FIG. 2. Accordingly, the real-time 
presentation of the annotation data is capable of being viewed by each participant of the 
videoconference session. (Paragraph [0068]). 

Claim 18, as amended, recites displaying a live video image, wherein the live video image 
shows a view of a location including one or more physical devices at the location; and annotating 
the view of the location shown in the live video image with an annotation. 

As described above, while Ludwig appears to show a teleconference system that includes 
live views of participants, each live view appears only to show a close-up of the face of the 
participant, i.e., none of the live views appear to show a view of a location including one or more 
physical devices at the location. Additionally, although Nelson appears to show an annotation 
management system that enables users to annotate a display region, in Nelson the display region 
does not appear to show a view of a location including one or more physical devices at the location. 
Instead, the display region in Nelson appears to show the content being annotated. As such, 
Applicant respectfully submits that Ludwig in view of Nelson does not appear to disclose or render 
obvious displaying a live video image, wherein the live video image shows a view of a location 
including one or more physical devices at the location; and annotating the view of the location 
shown in the live video image with an annotation, as recited by Claim 1, as amended. 

Claim 18, as amended, further recites automatically transferring the annotation to one of the 
physical devices at the location shown in the live video image if the annotation is at least partially 
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drawn over the physical device as it appears in the live video image. 

As described above, in Nelson the users appear to annotate a display region that is shown 
on a projector which appears to show the content being annotated. Additionally, annotations 
appear to be displayed via the projector regardless of whether the annotations are at least partially 
drawn over any physical device shown in a live video image. 

In view of the above comments, Applicant respectfully submits that Claim 18, as currently 
amended, is neither anticipated by nor obvious in view of the cited references, and reconsideration 
thereof is respectfully requested. 

Claims 19-23 

Claims 19-23 depend from and include all of the features of Claim 18. Claims 19-23 have 
not been addressed separately herein; however, Applicant respectfully submits that these claims 
are allowable at least as depending from an allowable independent claim, and further in view of the 
amendments to the independent claim, and the comments provided above. Reconsideration 
thereof is respectfully requested. 

Claims 24-32 and 34-35 

In the Office Action mailed February 9, 2010, Claims 24, 26-27, 30, 32, and 34-35 were 
rejected under 35 U.S.C. 103(a) as being unpatentable over Emens in view of Ayatsuka, and further 
in view of Burt ("Object tracking with a moving camera," IEEE An Application of Dynamic Motion 
Analysis, 1989, pp 2-12) and Nelson. Claim 25 was rejected under 35 U.S.C. 103(a) as being 
unpatentable over Emens in view of Ayatsuka and Burt, and further in view of Andersson (U.S. 
Patent Publication No. 2002/0111999 A1). Claim 28 was rejected under 35 U.S.C. 103(a) as being 
unpatentable over Emens in view of Ayatsuka, Burt, and Nelson, and further in view of Hildebrandt 
(U.S. Patent Publication No. 2004/0070616). Claim 29 was rejected under 35 U.S.C. 103(a) as 
being unpatentable over Emens in view of Ayatsuka, Burt, Nelson, and Andersson, and further in 
view of Westfield (U.S. Patent No. 6,677,979). Claim 31 was rejected under 35 U.S.C. 103(a) as 
being unpatentable over Emens in view of Ayatsuka, Burt, and Nelson, and further in view of 
Westfield. 

Claim 24 

The remarks provided above with regard to the rejection of Claim 10 under 35 U.S.C. 103(a) 
over Ludwig in view of Ayatsuka and Nelson are herein incorporated by reference. 
Claim 24, as amended, recites: 
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24. (Currently Amended) A shared interactive environment, comprising: 

a camera system to provide a first live view of a location and a second live view of 
the location, wherein the second live view can be configured to zoom in on a portion of the 
first live view, and wherein each live view shows a different view of a plurality of physical 
devices at the location; 

a first graphical user interface (GUI) coupled to the camera system which presents 
the first live view and the second live view, wherein each view shows one of more of the 
plurality of physical-devices at the location; 

a device controller to dynamically control the physical device in response to 
interaction of a first user with the GUI wherein the interaction can include annotating at least 
one of: 1 ) the first live view of the location; and 2) the second live view of the location; 

wherein annotations are automatically transferred to the physical device in the live 
views if the annotation is at least partially drawn over the physical device as it appears in the 
live video image, and wherein the annotation is displayed on the physical device such that 
the annotation can be viewed at the location; 

a device tracker coupled to the camera system and to dynamically recognize new 
physical devices; and 

wherein the camera system can be mounted on a mobile, robotic platform. 

Emens discloses a method for controlling a remote device from a client computer using a 
digital image of a remote location associated with the remote device. (Abstract). 

Ayatsuka discloses that "Hyper-drag" provides a user interface environment for the portable 
user computer to cooperate with a shared computer designed to use a desktop or a wall as display 
screen. For example, "hyper-drag" operations can be supported by introducing a camera-based 
object recognition system into the information environment. More specifically, the image of a 
computer display screen may be projected onto the surface of a disk top [sic] or a wall by means of 
a projector. Furthermore, both a projected image that is an object in the logical space of the 
computer and an object of the real world (that may be found on a disk top [sic]) may be recognized 
on the basis of a corresponding image picked-up by a camera and the motions of the objects may 
be traced so that the user can operate the computer interactively on the display screen that is now 
expanded to the disk top [sic] or the wall. (Column 3, lines 49-63). If, on the other hand, the target 
is a device that can display transparencies, the transparency is transferred to the projector (Step 
S107) in response to the request of the user of the user terminal 10 for transferring the current 
transparency to the projector (Step S106). A request for transferring a transparency may be issued 
in an intuitive way by using, for instance, "hyper-drag" and expanding the current GUI operation into 
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the space of the real world. Transparencies are transferred by way of the network 70. After the 
completion of the transfer of data, the processing operation returns to Step S101 and repeats the 
above steps. (Column 21, lines 7-17). 

Burt discloses that the task of detecting and tracking moving objects is particularly 
challenging if it must be performed with a camera that is itself moving. Yet, in applications such as 
automated surveillance and navigation, this task must be performed continuously, in real time, and 
using only modest computing hardware. (Abstract). 

Nelson discloses an annotation management system for providing real-time annotations for 
media content during a videoconference session. (Abstract). FIG. 8 is a simplified schematic 
diagram of a conference room configuration in which video conference participants view a video 
conference session from a liquid crystal display (LCD) projector in accordance with one 
embodiment of the invention. Here, in order to support annotation capabilities for participants in the 
conference room the system will support small devices, such as a pocket personal computer 
connected wirelessly to the network running a small device annotation client software. (Paragraph 
[0067]). FIG. 9 is a flow chart diagram illustrating the method operations for providing real-time 
annotation data to clients of a video conference session in accordance with one embodiment of the 
invention. The method initiates with operation 220 where a display region of a user interface 
associated with a client of the video conference session is annotated. Here, a participant of the 
video conference session may annotate a display region through the use of a mouse, stylus, or 
some other input device in order to highlight, distinguish or somehow otherwise annotate the 
display region. The method then advances to operation 222 where the annotation of the display 
region is detected. For example, a client monitor or some similar functionality, as mentioned above, 
may detect the annotation of the display region. The method then proceeds to operation 224 where 
in response to detecting the annotation of the display region, data corresponding to the annotation 
of the display region is communicated to other clients of the videoconference session. Here, the 
back channel as discussed with reference to FIG. 1, is used to communicate the annotation of the 
display region to the media transport server which in turn communicates the annotation data to the 
annotation management system described with reference to FIG. 2. Accordingly, the real-time 
presentation of the annotation data is capable of being viewed by each participant of the 
videoconference session. (Paragraph [0068]). 

Claim 24, as amended, recites a device controller to dynamically control the physical device 
in response to interaction of a first user with the GUI wherein the interaction can include annotating 
at least one of: 1) the first live view of the location; and 2) the second live view of the location; 
wherein annotations are automatically transferred to the physical device in the live views if the 
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annotation is at least partially drawn over the physical device as it appears in the live video image, 
and wherein the annotation is displayed on the physical device such that the annotation can be 
viewed at the location. 

In the Office Action, it was asserted that although Emens, Ayatsuka, and Burt do not 
disclose these features, Nelson does disclose these features. However, Applicant respectfully 
submits that Nelson does not appear to disclose that annotations are automatically transferred to 
the physical device in the live views if the annotation is at least partially drawn over the physical 
device as it appears in the live video image. Instead, as described above, in Nelson the users 
appear to annotate a display region that is shown on a projector which appears to show the content 
being annotated, which does not appear to include physical devices. These annotations then 
appear to be displayed via the projector regardless of whether the annotations are at least partially 
drawn over any physical device shown in a live video image. As such, Applicant respectfully 
submits that Emens, in view of Ayatsuka, Burt and Nelson, does not appear to disclose or render 
obvious the embodiment of Claim 24, as amended. 

In view of the above comments, Applicant respectfully submits that Claim 24, as currently 
amended, is neither anticipated by nor obvious in view of the cited references, and reconsideration 
thereof is respectfully requested. 

Claims 25-32 and 34-35 

Claims 25-32 and 34-35 depend from and include all of the features of Claim 24. Claims 25- 
32 and 34-35 have not been addressed separately herein; however, Applicant respectfully submits 
that these claims are allowable at least as depending from an allowable independent claim, and 
further in view of the amendments to the independent claim, and the comments provided above. 
Reconsideration thereof is respectfully requested. 

IV. Request for Interview 

In the event the above remarks fail to place the case in condition for allowance, Applicant 
respectfully requests the opportunity to interview with the Examiner at her convenience, and prior to 
the issuance of a subsequent Office Action, to assist in expediting prosecution. The Examiner is 
invited to telephone the undersigned if he can assist in any way in expediting issuance of a patent. 

V. Conclusion 

In view of the above amendments and remarks set forth above, it is respectfully submitted 
that all of the claims now pending in the subject patent application should be allowable, and 
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reconsideration thereof is respectfully requested. The Examiner is respectfully requested to 
telephone the undersigned if he can assist in any way in expediting issuance of a patent. 

A Petition for Extension of Time is submitted herewith, together with the appropriate fee. 
The Commissioner is authorized to charge any underpayment or credit any overpayment to Deposit 
Account No. 06-1325 for any matter in connection with this response, including any fee for 
extension of time, which may be required. 

Respectfully submitted, 



Date: July 9. 2010 By: /Nathan L Feld/ 

Nathan L. Feld 
Reg. No. 59,725 



Customer No. 23910 
FLIESLER MEYER LLP 
650 California Street, 14 th Floor 
San Francisco, California 94108 
Telephone: (415)362-3800 
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